Memory management, especially managing space in a memory, for example, when creating a snapshot for a file system or a virtual machine system, has gained prominence in current times. It is generally known to those skilled in the art that snapshot is a technique for system protection, which may save all states, data, and context of a system at any given instant of time. Snapshot operation is therefore critical to fault recovery of a system. For example, when a system has encountered an error during operation, it may be appropriate to recover it to the state and context at the time of creating the snapshot.
For example, consider a file system, a common snapshot created runs in at a volume level of the file system and manages snapshot relationship (i.e., metadata of the snapshot) at block granularity. Typically could be many snapshots (up to thousands of levels) in a file system, which could consume a considerable amount of memory to store metadata. For example, one snapshot of a 2 TB production file system could theoretically consume easily about 4 GB memory to store the metadata. Thus, in general the snapshot metadata are not completely stored in the memory, but might be “swapped in” and/or “swapped out” dynamically at runtime from an external memory device such as a disk. Alternatively, snapshots in a system may share a global memory space quota, but not all snapshots are equal, and some snapshots (generally the most recent snapshots) are considered to be more important and critical than the other snapshots. Such instances lead to several challenges especially for management of memory space. In order to guarantee an overall efficiency and performance (typically to reduce high CPU consumption), it may be desirable for managing memory space more efficiently.
Traditionally two common management approaches are known for memory management of file system snapshots:
1) all snapshot file systems share a single global memory management application (for example, paging daemon), and the pages in the memory are simply swapped according to the Least Recently Used (LRU) rule; or
2) each file system has its own memory space management application and manages the memory space independent of each other.
For the first approach, a single paging process is likely to be a bottleneck, generally in a large scale environment. The extensive IO (input/output) workload is likely to cause the memory space to reach an upper threshold of the quota (high watermark). Besides, this approach lacks in capabilities with respect to Quality of Service (QoS) management. For the second approach, without sufficient coordination between file systems, a signal paging process will encounter the case of dramatic jitter of the memory utilization rate. For example, too many blocks might be swapped out in parallel by the paging daemon, and then immediately swap in may be requested by coming reads, which tends to causes too many disk I/Os and thereby lowers the swap efficiency. Besides, another challenge lies with the difficult to work out an efficient, definite and luminous rule to divide out budgets among file systems and enable QoS.
In summary, in the traditional memory space management methods, the swap in/swap out decisions of pages in the memory are made without knowing the overall system state. It is sometimes not possible to guarantee that a relatively high importance file system has a corresponding priority consideration in the memory management. Meanwhile, in traditional implementation, for sake of simplicity, typically all snapshot metadata share a global LRU list and memory page management is executed by a single thread, which is likely cause a bottleneck and result in high CPU consumption and lacks scalability. These drawbacks exist in other fields such as virtual machine memory management as well.